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Decision feedback equalization is presently of interest as a technique 
for reducing intersymbol interference in high-rate PAM data communica- 
tions syste?ns. The basic principle is to cancel out intersymbol interference 
arising from previously decided data symbols at the receiver, leaving re- 
maining intersymbol interference components to be handled by linear 
equalization. In this work we consider the application of decision feedback 
equalization to quadrature-amplitude modulation (QAM) transmission, in 
which two independent information streams modulate quadrature carriers. 
Extending Salz's treatment in a companion paper of decision feedback for 
a baseband channel, ive derive the form of the optimum receiver filters via a 
matrix Wiener-Hopf analysis. We obtain explicit analytical expressions 
for minimum mean-square error and optimum transmitting filters. The 
optimization is subject to a constraint on the transmitted signal power and 
assumes no prior decision errors. The class of QAM transmitter and re- 
ceiver structures treated here is actually much larger than the class usually 
considered for QAM systems. However, our results for decision feedback 
equalization show that, for nonexcess bandwidth systems, optimum per- 
formance is achievable without taking advantage of the most general struc- 
ture. If the transmitter is required to have the conventional QAM structure, 
study of the time continuous system that gives rise to the sampled data sys- 
tem considered here demonstrates that under quite general assumptions a 
nonexcess bandwidth system is optimum. Finally, the explicit description 
of the optimum transmitting matrix filter follows from an information- 
theoretic "water-pouring" algorithm in conjunction with the determination 
of the form of the points of maxima of a determinant extremal problem. 

I. INTRODUCTION 

Interest has recent]}' intensified in receiver structures which hope- 
fully will permit higher data symbol rates than are possible with con- 
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ventional demodulator/linear equalizer structures having the same 
error probability. The decision feedback equalizer is an example of a 
receiver component that can have important performance advantages 
over a linear equalizer operating over dispersive channels with additive 
noise. 1-7 The basic structure of a decision feedback equalizer (DFE) 
is shown in Fig. 1 . The function of the filter in the feedback path is to 
cancel "postcursors" of the channel's impulse response; that is, inter- 
symbol interference components arising from previously decided sym- 
bols. Thus, the job of the linear filter in the forward path is to minimize 
(according to some criterion) ' 'precursors" of the channel's impulse 
response which cause intersymbol interference from future data sym- 
bols. Of course, there is a possibility of error propagation with this 
nonlinear feedback structure. We avoid this intractable problem by 
assuming that no erroneous decisions pass into the feedback filter. 
Thus, our results provide a performance lower bound. Earlier experi- 
mental studies indicated that error propagation is not a serious problem 
on some channels. 3,4 

Price 6 (whose bibliography on the subject is extensive) has derived 
asymptotic formulas (allowing for an infinite number of equalizer taps) 
for error probability, optimum transmitter pulse spectrum, and com- 
munication efficiency for the "zero-forcing" DFE, which minimizes the 
noise variance at the DFE output subject to the constraint that the 
intersymbol interference is zero at the receiver's sampling instants. As 
is the case for linear equalization, the mean-square-error (MSE) cri- 
terion is more general than the zero-forcing criterion. The MSE cri- 
terion minimizes the mean square of the total error (noise plus residual 
intersymbol interference) at the DFE output. 2,5 Asymptotic results 
and illuminating calculations of performance for MSE-minimizing 
DFE's are contained in a companion paper by Salz. 7 

All previous theoretical studies of decision feedback equalization 
have assumed a "baseband" linear PAM channel model depicted in 
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Fig. 1 — Basic decision feedback equalizer structure. 
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Fig. 2 — -Baseband channel model. 



Fig. 2. The transmitted waveform s(t) is 

s (6) (0 = Za n g^(t-nT), 

where the data symbols {a n \ are statistically independent discrete- 
valued random variables from a finite set and g ib) (t) is some suitable 
transmitted pulse waveform. The channel output waveform is then 

r<»>(0 = £ a n h^(t - nT) + «<»(*), 

n 

where the overall impulse response is 



h w 



/DC 
-o 



C (6)( r ) & (6)( < _ r ) dj . 



and n ib) (t) is additive noise. This model is certainly valid for a real 
linear channel accepting ever}' T second a pulse of the form a„g (b) (t). 
It is also valid for the important case where the linear channel c (6) (f) is 
actually the baseband equivalent of a passband channel when the 
modulation is either double-, vestigial-, or single-sideband. (See Ref. 
8, Chapter 7.) Of course, c ft) (0 then depends on the carrier frequency 
and on any phase offset between the reference carriers at the modulator 
and demodulator. 

In this paper we extend the asymptotic DFE theory to the case of 
QAM (quadrature amplitude modulation) signaling, for which the 
baseband model of Fig. 2 is not sufficient. We summarize our results 
at the end of Section II. The most general QAM transmitter structure 
is illustrated in Fig. 3. Two independent data sequences enter a lattice 
network comprising filters with impulse responses gu(t), Qzi(t), 0is(O> 
and 022 (0- Modulation is done with two quadrature carriers with fre- 
quency /o Hz. In practice, most QAM transmitters are specialized to 
the case Qu(t) = gii{t); Qn(t) = —gn(t).' We call the class of trans- 



* Indeed, it is often assumed that gn(t) and gn(t) are zero. 
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sin 2 7T f n t 



Fig. 3 — General QAM transmitter structure. 



mitters with this special structure the class of "passband" transmitters 
((?). We show in later sections that optimum performance is in general 
achievable by restricting the transmitter to this class or a simple 
variant thereof. It is worth noting that QAM systems with passband 
transmitters are mathematically equivalent to baseband PAM sys- 
tems, but with complex impulse responses and information symbols. 9 - 10 
For the most general QAM structure, the waveform s(t) is expressed 
in terms of two-dimensional vectors and matrices. Define the vector 
a„ to be the nth. pair of information symbols, 



'•=(£)• 



(l) 



The most general QAM transmitter is characterized by the matrix 

filter 



9{) wo *»(«)/• 



Then the structure of Fig. 3 yields 

s(t) = (cos 2ir/rf, sin 2irf t) £ g(t - nT)a„. 



(2) 



(3) 



We assume that the data symbols are uncorrelated discrete-valued 
random variables with variance a\. Thus 



(a„at.) = crldnml, 



(4) 



where ( ) denotes expectation, f denotes transpose,* 8 nm is the Kro- 
necker delta, and / is the identity matrix. The transmitted power is 



* The symbol f will denote conjugate transpose for complex vectors and matrices. 
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then given by 

P = lim I- JT ( S *(t))dt = ||£ Q&(0 + <&(*) + & (0+ gkffldt 

= 4^/_" [|^'ii(^)l 2 + |Gi 2 (co)| 2 + |(?2i(co)| 2 + |G' 22 (o»)| 2 ]dco,(5) 

where O\,(co) is the Fourier transform of g»y(<)- For future reference 
note that we can also write P as 

T Kr2ll OS H yrj + 0-221 W H m - I »W 



where 



G(«) = ( Gu(w) Gl,(w) ^ 

\ 0^21(0)) Gm(u)) } 



is the matrix frequency response of the transmitter. We use tr to denote 
the trace of a matrix. 

Later sections will show that without an initial assumption of the 
special passband transmitter structure the treatment of decision feed- 
back equalization for two- (and hence higher) dimensional signals is a 
nontrivial generalization of the baseband signal case. 

II. THE CHANNEL MODEL AND SUMMARY OF RESULTS 

The impulse response q(t) of any linear channel can be resolved about 
a center frequency / : 

q(t) = ci(t) cos 2vfot - c 2 (t) sin 2tt/ «. (7) 

It is easy to show that the channel model of Fig. 4 yields exactly the 
above impulse response, and thus any linear channel can be conve- 
niently represented in terms of an arbitrary center frequency / by the 
structure of Fig. 4. We note in passing that the so-called "in-phase" 
and "quadrature" impulse responses Ci(t) and d{t) are often inter- 
preted as the real and imaginary parts, respectively, of the "complex 
envelope" of the impulse response q(t) with respect to the frequency 

u 

We assume that the low-pass transmitter impulse responses iga(t)\ 
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Fig. 4 — A passband channel model. 



are all strictly bandlimited to lie within the frequency interval ( — / , /o) ; 
otherwise, the system would suffer distortion from aliasing effects. 
There is then no loss of generality in assuming that the channel's in- 
phase and quadrature impulse responses ci(t) and ci{t) are also strictly 
bandlimited to this interval. 

With these assumptions, double-frequency terms disappear, 11 and it 
is easily shown that the noise-free channel output 



/.' 



q(T)s(t - r)dr, 



where q(t) is given by (7) and s(t) by (3), can be written 

£(cos 2irf t, sin 2tt/oO £ / c(t — r)g(r — nT)d,Ta n , 

n J — oo 

where c(t) is the matrix 

c(t) = ( Cl (0 *(0\ 

CW \-c 2 (t) Ci (t))> 



(8) 

(9) 
(10) 



the matrix g(t) is given by (2), and integration of matrices and vectors 
means integration of each entry. 

Consider receiver structures whose "front end" is the type shown in 
Fig. 5 — sine and cosine demodulators followed by identical ideal low- 
pass filters that are strictly bandlimited to ( — /o, /o) and whose outputs 
are labelled r(t) and f(t), respectively. This structure causes no loss of 
information, since any bandlimited input signal can be reproduced 
exactly if the outputs r(t) and r(t) are multiplied by cos 2irfat and 
sin 2rfot, respectively, and then added together. The function of the 
low-pass filters is to remove double frequency terms ; it will turn out 
that the "front end" will be followed by a band-limiting matched filter, 
so that the low-pass filters are not necessary. 
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Fig. 5 — Receiver "front end." 



The low-pass outputs r(t) and f(t) can be written in vector form as 
r(t) =(^)= \ZKt - nT)a n + v(0, (11) 



where the matrix impulse response h (t) is 



h 



(t) = f_ 



c(t)o(t - r)dr, 



(12) 



and the components of the vector 

v(0 = 



(n c {t)\ 
WO/ 



represent additive noise. Assuming that the additive noise in the 
channel is white with double-sided power spectral density No/2, it can 
be shown 11 that n c (t) and n a (t) are statistically independent stationary 
zero mean processes; each is the result of passing a stationary white 
noise with double-sided power spectral density No through an ideal 
low-pass filter. Noise outside the signal bandwidth will be eliminated 
by a matched filter. Accordingly, we take the covariance matrix of the 
noise to be 

(v(t)vHt + r)> = NoIS(t), (13) 

where / is the identity matrix and 8(t) is a "unit-area delta function." 
The mathematical model for the transmitter and channel is now com- 
plete and is summarized in Fig. 6a. 

We remark in passing that linear modulation of a single stream of 
data symbols (e.g., single-sideband or vestigial-sideband modulation) 
constitutes a special case of this model. In that case, gn{t) = ffas(0 — 0, 
and the receiver front end consists of a cosine demodulator with some 
phase shift 6, followed by an ideal low-pass filter. Then the overall 
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Fig. 6 — (a) Canonical mathematical model of transmitter and channel, (b) QAM 
decision feedback equalizer structure, (c) Structure of the matrix filter w(t). 
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impulse response is a scalar function of time (which depends on the 
receiver phase shift 0), and hence all two-dimensional matrices and 
vectors in the present treatment would be replaced by scalar quantities 
(see Ref. 7). 
The following list summarizes our main results: 

(i) The optimum linear forward filter at the receiver for a given trans- 
mitter channel cascade, H(co) = C(co)G(co), is found to have the form 

Const X H(o)/ x k(<r* T ) +^?J , 



where 



and V** denotes minimum-phase square root. This filter can be viewed 
as a matrix matched filter followed by an anticausal matrix tap delay 
line. (See Sections III and VI.) 

(ii) For a given transmitter power spectral density: if a nonexcess 
bandwidth system (Section V) is required, an optimum transmitter is 
found and it is passband ; conversely if the transmitter is taken to be 
passband, the optimum system is a nonexcess bandwidth one (Sec- 
tion III). 

(Hi) Given a passband transmitter, the MSE (the sum of the mean- 
square errors of the two unquantized receiver outputs) is 



2 ^{-lJZ l0 im Xe * ( " 



) + i 



do 



where 



x|fc(-+7 ! )+*fc(«+T ! )r 

and Gi(co) = G n (o>) = G-nfa) and Gi(u>) = Gu(co) = — G 2 i(u) (Sec- 
tion VI). 

(iv) The optimum transmitter power spectral density is found for the 
class of passband transmitters meeting an output power constraint 
(Section VII). This optimal density has a water-pouring description. 
(Since the processing capability considered here represents an advance- 
ment over conventional linear equalization, this emergence of an in- 
formation theoretic type density is perhaps not surprising.) 
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(v) Although we do not constrain the in-phase and quadrature mean- 
square errors to be equal, we show that for the above-mentioned opti- 
mal systems the errors on the two data streams have equal variances 
and are uncorrelated (Section VI). 

In a nutshell, the system optimization proceeds as follows : 

(i) Find the optimal receiver for each transmitter. 
(ii) Find an optimal transmitter for each transmitter power spec- 
tral density. 
(Hi) Find the optimal transmitter power spectral density. 

Then we reverse, using the solution of (Hi) to specify an optimal trans- 
mitter and then using this optimal transmitter to specify the optimal 
receiver. 

III. THE RECEIVER OPTIMIZATION PROBLEM 

The DFE structure consists of a linear matrix filter w(t), quantizer, 
and a transversal feedback filter with matrix tap coefficients {&„} which 
processes previously made decisions as shown in Fig. 6b. The kth. 
sampled vector input to the quantizers is written 



Yk 



= /"" w(r)i(kT - r)dr - £ 6„a fc _„, (14) 

J-v> n = l 



where a„ is the receiver's decision on the nth. data symbol-pair. Note 
that we allow the feedforward and feedback matrix filters to have 
infinite-duration impulse responses. We also replace at_„ in (14) by 
the true data symbol vector a*_„ for mathematical tractibility ; thus, 
we in effect postulate a "magic genie" preceding the feedback filter 
who corrects any decision errors. The genie's existence is immaterial 
up to the time of the first decision error, and hence our expression for 
MSE is certainly valid up to that time. 

The error vector e„ is defined to be the difference between y* and the 
correct symbol a*, 

£k = Vfc - a*, (15) 

and the MSE is defined to be the trace of the error matrix e , where 

eo = <e n 4>, (16) 

the average being with respect to the noise and the data symbol se- 
quence. Note that e is positive semidefinite and symmetric. 

Substituting (14) and (15) into (16) and using the noise correlation 
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matrix (13) and the data symbol correlation matrix (4), we can write 
e = al £ f* f w(n)hl(k - n)T - r{]tf[{k - n)T - r 2 ] 

n^k J — oo J — oo 

X w; t (T 2 )dTidr 2 + N J J if (ri)«; t (r2)5(T 1 - T 2 )dTid T2 

+ cr§7 + a\ £ r&„ - /"" w( T )h(nT - t)cIt~\ 

X b n - J" w(r)h(nT - T )dr\ - cr* /"" iv( T )h(-T)dT 

-<&(" hH-T)wH T )dT. (17) 

y— 00 

We immediately observe that tr e is minimized with respect to the 
matrices {&„} if and only if for all n > 1, b n = s„, where 

s n = f w{ T )h(nT - T )dr all n (18) 

represents the matrix samples for n ^ 1 of the impulse response of the 
transmitter /channel-receiver filter combination. Then, once the {&„} 
are optimized in this way, the remaining terms comprising the matrix e 
can be written 

e - <rl £ |"*„o/ ~ f* w(r)h(nT - r)dr I 

ngO L •/-» J 

X 8„o/ - /" w( T )h(nT - T )dr\ + iV„ /"" 10(7)1^ (t^t. (19) 

■/ —00 J ■/ —00 

We wish to minimize tr eo with respect to the entries in the matrix 
w(t). Notice from eq. (16) that tr e is a positive quadratic form. Thus 
from Ref. 12 we set the gradient equal to zero to determine the sta- 
tionary points which are necessarily points of global minima. We shall 
find that there is only one solution. 

Proceeding with the calculus of variations method, we replace 

w(t) = ( w n(t) Wm(0\ 

by 

\ 6211721(1) e 2 2l?22(U/ 
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where the Wj{t) are arbitrary. Setting 



dtr e dtr er 



den d«i2 
dtr eg dtr eo 



de? 



de? 



-G3 



at Cll = 6i2 = «21 = «22 = 0, 



we get 



-*W(-t)+o* L f" v>Mh(nT - rJdnhHnT - r) 

+ iVow(r) = [0] 



or 



where 



w(t) = £ w n W{nT - t), 

n<0 



w n = ^ (8 n0 I - s„) n ^ 0. 
ivo 



(20) 



(21) 



This means that the matrix filter w(0 can be interpreted as a matrix 
matched filter with impulse response /i f ( — t ) followed by a sampler and 
matrix transversal filter with matrix tap coefficients {w„\. Note that 
the transversal filter is "anticausal" — that is, w n = [0] for n > 0. The 
structure of w(t) is illustrated in Fig. 6c. 

Furthermore, substitution of the optimum filter (10) back into ex- 
pression (19) for the error matrix eo results in 

e = <tI(I - s o y (22a) 

and from (21) 

e = Nowt (22b) 

An explicit solution for the optimum tap coefficient matrices {w n \ 
can be obtained by postmultiplying (20) by h(mT — t) and integrat- 
ing, using (21) and (18) and the definition 



to yield 



,p n = r h\-r)h{nT - t)cLt 

J — CO 

E w m n _ m + -§ 8 m - n , I = S„o7 for n ^ 0. 

fi£0 . 0"a J 



(23) 



We recognize eq. (23) as a classical Wiener-Hopf equation for which 
we are assured the existence of a unique solution. 13 

We attach a plus (minus) subscript to any matrix sequence whose 
value is the zero matrix on the strictly negative (positive) integers.* 



* A matrix sequence u+, zero on the strictly negative integers, is referred to as 
causal. A sequence «_, zero on the strictly positive integers, is referred to as anticausal. 
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By 1 we mean the matrix sequence vanishing everywhere but . zero, 
where the value is /. Then (23) is written 

M>_*^= 1 (n ^ 0), (24) 

where we have used t to denote the sequence sum which we observe is 
the Fourier coefficient sequence for a positive definite Hermitian matrix 
function whose determinant is uniformly above (No/<rl). Hence j ad- 
mits a causal, anticausal decon volution of the kind provided by Wiener 
and Akutowicz 14 (generalizing a result of Szego). Based on Ref. 14, we 
can say 

A = U.*U + , { (it.), - ZM-nJK-O, (25) 

where we have used (w_)„ to denote the nth entry in the w_ sequence 
(similarly for u+). Corresponding to w_, we have its convolution in- 
verse, [u-Jr l , which is also anticausal. 
From what we have just said, 

C(w + )o]- 1 [w-]- 1 *«-V = 1, (26) 

and so w- = C(w + )o3 _1 ( w -) -1 an d, m particular, 

w = (ttfliij)- 1 , (27) 

where in the last equation the negative subscripts have been sup- 
pressed. Thus the anticausal transversal filter is found from eqs. (23) 
through (26) to have a frequency response inversely proportional to 



4 






where the notation V means minimum-phase square root and 
$(er iuT ) is the discrete Fourier transform of the matrix sequence {<£„}. 
Recalling eq. (22b), we have the following expression for the error 
matrix : 

e - NM = N£u<fljiR-K (28) 

We remark that the development so far is analogous to that of the 
baseband decision feedback equalizer. 5 Further progress toward achiev- 
ing a closed-form expression for tr e thus depends on obtaining a 
closed-form expression for the matrix \_u a uX]~ v or for its trace, corre- 
sponding to the result recently developed for the baseband case. 7 It 
has not been possible to do this directly for the QAM case when the 
most general transmitter matrix is allowed. We shall prove under quite 
general conditions that the minimum of tr wo is achieved with a trans- 
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mitter of passband structure, and that, given this transmitter struc- 
ture, 

tr iv o = ^Vdet w 
and 



- £■ [ VT logdet \*((r'» T ) + ^/W, 



det Wo = exp 
where "det" denotes the determinant. 

IV. CLOSED-FORM EXPRESSION FOR DET e 

For the most general matrix filter, attainment of a closed-form MSE 
expression for tr e in terms of the matrix $(e-'» r ) has so far proved 
intractable. However, we shall see that such a general expression is 
unnecessary to describe the behavior of optimum systems. Our ap- 
proach is to employ the following easily proven lower bound for 2 X 2 
positive semi-definite symmetric matrices 

trwo ^ 2|det*Wo|f (29a) 

which holds with equality if and only if io is a scalar matrix (i.e., 
multiple of the identity). In this section we develop a closed-form 
expression for det io . In the following sections where we deal with 
optimum systems, we can always perform the analysis in a context 
where eq. (29) holds the equality. 

We begin the analysis of Vdet io by recalling (25a), from which 
follows 

det [*(«r*»*) + ^I~\ = det U-{<r*" T ) det U -.{**)• 

Then from the one-dimensional theory we have 15 

det (uoul) = exp |£ ["* logdet [•(«-*") + ^ l]do 

[SIT J-rjT L °"« J 

and from (28) and (29a) 



^ tr e = tr w ^ 2 Vdet w , (29b) 

iVo 



where 



det ic = exp j - ^ f' T r log det [•(•"*") + ^ / W ■ (29c) 
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V. POIt A NONEXCESS BANDWIDTH SYSTEM, PASSBAND TRANSMITTERS 
CANNOT BE OUTPERFORMED 

In this section we begin by expressing <£ explicitly in terms of the 
transmitter and channel matrices. Then we define the notion of a 
nonexcess bandwidth system. The primary result of this section is that, 
for a nonexcess bandwidth system, if the transmitter power density 
function 

/ 1 r*' T 2TP\ 

/(») = fcr G*G I /(«) £ 0, i / /(«)<&, = t£f) 

is specified, then there exists a passband transmitter in the class of all 
matrix transmitters optimal under the constraint that /(w) is the power 
density function. 

To display the dependence of our results so far on the transmitter 
frequency response G(co), we first rewrite the matrix $(e~ i ' aT ) using the 
definition of 0„ as 

*(«-*•*) = r At(-r)3C(cu, r)dr, (30a) 

J — 00 

where 

3C(«, r) = £ fe(nT - r)e-^» r . (30b) 

Expression (30b) is a Fourier series. Thus, 

h(nT - t) = £- f' ' 3C(w, T)e*»* T du. (31) 

But the matrix impulse response h(nT — r) can also be written as 
the inverse Fourier transform of a matrix frequency response H(u>), 

h(nT - t) = jl C° H{u)e^ nT - r hlu>, 

-IT J-x 

which, upon splitting up the range of integration and changing the 
variable of integration, can be written 

1 f TlT 
h(nT - t) = / e*°* T 

SIT J-r/T 

x[Eff(» + ^)exp[- J ( M + ?^)r]]^. (32) 
Equating the integrands in (31) and (32), we obtain an explicit ex- 
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pression for 3C(w, t) which when substituted into (30a) yields 

•^-•^i: *(. + ??)«(. + ??)■ (33) 

Furthermore, denoting the Fourier transforms of c (t) and g (t) by the 
channel matrix C (w) and the transmitter matrix G (a>) respectively, we 
can write H{o>) = C(a>)G(u) and 

X c(„ + ^) + c(„ + 2 -^)G(. + *?) • (34) 

A nonexcess bandwidth system is defined by the property that for any 
radian frequency u there is no more than one nonzero term in the above 
sum. It can be taken to be the n = term by making a trivial fre- 
quency translation where necessary. Hence for a nonexcess bandwidth 
system 

•(«-*") - |Gf(«) t C(«)t(7(«)fl'(«) (|«| ^ |)- (35) 

In this section we deal exclusively with nonexcess bandwidth systems. 
In Section VIII we refer to a recent theorem of H. Witsenhausen which 
enables us to do a complete analysis of excess bandwidth systems by 
transforming them to a canonical nonexcess bandwidth "equivalent" 
and then transforming back. 

To model the class of transmitter frequency responses G(u), we intro- 
duce 9 to denote the (Hilbert) space of all 2 X 2 matrices whose entries 
are Hermitian symmetric {(?(«) = [<?( — &>)]*} finite energy functions 
on ( — tt/T, 7I-/7 7 ). The Hermitian symmetry of the entries is required 
so that each entry represents the Fourier transform of a real-time 
function. As in Section I, we use (P to denote the passband subspace of 
9 consisting of matrices of the form 



/6u(«) 6i,(«)\ 



We shall be dealing only with matrix filters G of finite power P, given 
by (6). Thus we use 9p and G> P to denote 



Gfi- f" tr G(«)(?(w) t dw - 

\2t J -tt/T 
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in g and (P, respectively. In the sequel, all transmitter filters will be 
assumed to have power P. 

We now optimize det (G^CK'G + (N /<rl) I) at each radian frequency 
o> for a fixed amount of power transmitted at w. 

Fix N > 0, / > and C = (j c C Q 2 \ (C?s complex functions 
of frequency). Explicitly, we shall show that 

max det \gWCG + ^I 



over all complex G such that tr G f G = /(«) is achieved for a G of the 

(G C \ 

— C CI' ^ For ^ near QAM systems, the same de- 
terminant extremal problem arises in the optimum selection of a trans- 
mitter with a specified power spectral density function. To our knowl- 
edge, this aspect of linear QAM systems has escaped the literature.) 

Notice the unitary transformation \F = 2~m . J diagonalizes 

matrices of the form ( ., J in that 

\ -jb a J 



\-jb ar \ a-b) 



(36) 



Since C*C is of the form ( _ a jb \ (a, 6, real, a > 6), if we let G = *B 
the problem becomes 

max det W a + 6 ° )tf + Nj\, tr B*B - /(«). 

Let D = ( _ , J and rewrite the problem as 

max {det (B f DB) + N tr (BWB) + NZ\, tr B*B = /(«). 

At this stage we denote the Hermitian matrix BB* by Q and write 

max {detQD + N tvQD + #§}, trQ = /(«). 

Of course, an optimum Q exists, since we are maximizing a con- 
tinuous function over a compact set. A nonzero off-diagonal entry in Q 
would only affect the determinant and not the traces. Since 
Q is Hermitian, the optimal Q is diagonal. Retracking, Q = BB* 
= ^GiW^G^ty. Now Q is positive definite and so has a positive 



1S38 THE BELL SYSTEM TECHNICAL JOURNAL, DECEMBER 1973 

definite square root QK From the definition of B, 

G = *QW 

which shows that the optimal G has the form 

Although the proof is now complete, we go further and find Gu and 
G12, as this will be used in the sequel. To find G n and Gu we have seen 
that we must first find the entries gu and g 2 2 of Q so as to maximize 

{qnqn(a + 6) (a - 6) + N \jm(a + b) + g 22 (a - 6)] + N 2 ] 

on the triangle in the (gn, $22) plane described by 

qn + 922 ^ /, 9ii ^ 0, g 22 ^ 0. 

Since a > and a ^ b, the optimum (511,522) is achieved with gu 
+ g 22 = /• Let X linearly parametrize the segment joining (/, 0) and 
(0, /) as shown below 

f22f 

solution on 
X = 1 this segment 



-> 



/ 5n 

so (g u , g 22 ) = [(1 - A)/, A/] where (0 ^ X =S 1). 
The criterion becomes: Maximize 

/ [X(l -X)/(a 2 - 6 2 ) + JVo[(l -X)(a + b) + X(a - &)]} + N 2 , (38) 

which is a parabola concave in X. Our problem is to determine 
X op t(0 ^ X opt ^ 1). Now the parabola is maximized at 



x _ i _ E»( h \ 

A 2 / \a 2 - &V 



If X satisfies < X < 1, then X opt = X. If X < 0, X opt - and if 

X > 1, X op t = 1. 

So from G = ^QW, we obtain 



Gn = [|V(1 -x,pt)| + |vx^|]|/*l (39a) 



Gi 2 = j d^i — x ~i — 1 vx^ t |)i/M signum b (39b) 



DFE IN QAM SYSTEMS 



1839 



The determination of the signs attached to G n and Gi 2 was made by- 
noticing that at each frequency 



det (gW*CG + ^ /) 



(40) 



is invariant to the sign of Gn, while (40) is maximized if signum b is 
used for Gu. 



VI. CLOSED FORM EXPRESSION FOR ALL PASSBAND G 

We have seen that, for nonexcess bandwidth systems, an extremal 
Gfor 



det w = exp 



T f' ,T T N n 

- 75- / log det *(«-*•»■) + S? J 



do 



exists in the space (P. Next we show that fo r each G£(P, whether or not 
it has excess bandwidth, tr w = 2Vdet iv . To do this we must show 
that wo is a scalar matrix. First observe that the matrices G and C are 
in (P, and their entries, being Fourier transforms of real-time functions, 
are Hermitian symmetric. The matrix $(e~' aT ) + (N /al)I, which is 
designated by (R and is the Fourier transform of the matrix sequence a 
in (24), can be expressed in terms of the channel matrix C(w) and a 
passband transmitter matrix 



as in (34) to yield 



where 



^ -(-£8 K) 

\-J«s(«) «■(«)/' 



«.(-)-^z[|o.(« + ^)c.(- + ^) 

+ ft (. + *«) ft (. + *?)|" + §] (40 
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and 

«,(-) = - | £ Im | [A (. + 2 -f) ft (. + ^?) 

+ (?1 ( M+ ^) C! („ + ^)]* 
x[ Gl ( M+ ^) Cl (« + ^) 

-A(.+*")C,(-+^?)]|. (42) 

The entries (Ri and (R 2 are real functions of w. (Ri is a positive even 
function and (R 2 is an odd function. The matrix (R is positive definite ; 
i.e., (R? > (Ri- It is also Hermitian and passband. 

We have previously noted in eq. (25) that the matrix (R can be 
factored into the anticausal and causal matrices U-(e~ iaT ) and 
[U _(e~ ju,T )y , respectively. The matrix u ul, which is proportional to 
the error matrix inverse, is unique and the factor U-(e- jwT ) is unique up 
to an arbitrary unitary matrix post-multiplicative factor SF. We now 
pick a particular unitary matrix. 

The matrix (R is diagonalized by the unitary matrix 

-4G 0' •*•*«- (*o* *+ J (43) 

Now the entries (Ri — (R 2 and (Ri + (R 2 are nonnegative real functions 
on -(ir/T) ^ a ^ (v/T). Since 

- oo < / log (<Ri ± (R 2 )dw, 

J -■kIT 

we have from Szego's theorem 15 that 

(Ri - (R 2 = |a-| 2 (44a) 

and 

(Ri + (Ri = |/3_| 2 , (44b) 

where a_ and /3_ are anticausal functions of w, i.e., 

a- = L a^-*»' (45a) 

TO gO 

and 

0_ = L m e~^ mT ; (45b) 

TO gO 

the {a m } and {ft m } being sequences of complex numbers. We can assume 
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a and O are real and positive without loss of generality. Therefore 

¥ f (R¥ = VV\ (46) 

where 

V = ( a ~ °) 

and since ^ is a unitary matrix, 

= U-UJ, (47a) 

where 

\-j(a--0.) a_ + /3_ J' (47b) 

Thus in this factorization, £/_ can be taken to be passband. Further- 
more, since a and /3o are real, 

W °-(-,(ao-/5o) ao + 3o ) (48) 

is both Hermitian and passband, and so therefore is the error matrix 
eo — [uoUo]- 1 ; i.e., its off-diagonal terms are purely imaginary.' But 
we know that e , defined by (15), must have real equal off-diagonal 
terms, and therefore e must bo a scalar matrix. Thus iv = (l/N )e is 
also scalar and 

tr wo — 2Vdet w . (49) 

Summarizing the development so far, we have shown that, for non- 
excess bandwidth systems, if the transmitted power spectrum is speci- 
fied, the passband transmitter structure is optimum. We then showed 
that if the transmitter has the passband structure, the MSE is given 
by eqs. (29b) and (29c), (29b) holding with equality. Incidentally, 
usin g the results of the last paragraph it can be shown that a = £o 
= VAfo/2MMSE ; hence u is known. In Section III the optimum linear 
receiver filter w{t) was found up to the constant (matrix) multiplier 
US' 1 . For nonexcess bandwidth passband systems, we can now make the 
more complete statement that the matrix Fourier transform of w{t) is 



4 



l -^ Gt(«)Ct(«)/^Gt(«)Ct(«)C(«)6(») 



12 



(where V * means minimum-phase square root). 

* It is important to notice that, although u is unique only up to a postmultiplica- 
tive unitary factor, the matrix u u is unique. 
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We shall now express (29c) in a somewhat different form, which 
avoids the use of the determinant. As pointed out earlier, flti(w) and 
(R 2 (w) are real even and odd functions, respectively. Thus 

(Ri(oj) - (R 2 (w) = (Ri(-w) + <R 2 (-«) 
and 

r' T log [(Ri(«) - m 2 (aj)]daj = /"" log [(Ri(«) + (Rg(w)]dw, (50) 

J-w/T J-rlT 

from which it follows that 

[T f rlT 1 

-— / logdet (R(w)dw 

= 2JV exp ( - ^ I J' log [<Ri(«) - (R 2 («)]dw 

+ /"" log[(Ri(w) + <Ra(w)]<M J 



= 2AT exp - ^ j' log [<Ri(«) + (R 2 («)]dw 



(51) 



Substituting (41) and (42) into (51) gives the following expression 

for MSE : 

tr e = 2al exp j - ^ £'* log [^ X c » + l]«fc} , (52) 
where 



xd, ,^) + i c 2 (. + ^ 



This expression is valid for any passband transmitter and, as shown 
in the previous section, it is valid for optimum general QAM trans- 
mitters with no excess bandwidth.* We show in Appendix A that, 
under very general assumptions, optimum passband transmitters will 
have no excess bandwidth. 



* We remark at this point that if we had restricted attention to passband trans- 
mitter structures from the outset, we could have derived the MSE expression (52) 
more directly by using the complex envelope notation referred to in Section II instead 
of the matrix formulation. 
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VII. OPTIMUM TRANSMITTER 

Here we continue under the assumption of a nonexcess bandwidth 
system. So far, we know that, if / = tr G f G is specified, an optimal 
passband G exists yielding a minimum for MSB and possessing power 
spectral density /. Our next step is to free tr G*G and to find G, which 
minimizes MSE subject only to the constraint that 



T f VIT / T r r ' T \ 

h f=(h trGft0-)-P. 

l-K J-t/T \-ilT J-r/T J 



(53) 



Notice 

'IT rr/T 



f |CFi + jO,|>- f" (|Gi|» + |G 2 | 2 ) (54) 

J-r/T J-r/T 

since G\G 2 — GiG* 2 is odd. Thus our problem becomes to find 
|Cri + i^l 2 , minimizing 

^-|^>(° , ' |Cl + ^ ,+iC ' l ' + 1 ) 

71 rr/T 

subject to jf- / \Q t + j'G 2 | 2 = P. 

It is shown in Appendix B that the solution to this problem is given 
uniquely by 

Id + jG t \* = (e - ^ |d + jC 2 |- 2 ) 



+ 

(where (*)+ = maxCfcO)]), 



where G is a constant set at a value so that 

7 1 rrlT 

'-ir/r 



T r*iT 

f / (|Gl|» + [(? 2 | 2 ) =P. 

Z7T J-t/T 



This solution also occurs in a related context in information theory, 
where it is dubbed "the water-pouring solution." 16 

Since (G*(<a)G 2 (u) — Gi(o))G* 2 (w)) is odd and /(w) is even, we average 
!<£(«) + jG 2 (u>)| 2 and |(?t(-«) + jQ t {-u)\* to gef 



/(-) = i 



A^n!T 2 , , 1 

+ [c - -^p |<m-«) + yc 2 (-«)|- 2 jj 



To find Gi(co) and (? 2 (a>), use the above /(co) in Section V. 



* Note that for jVo — * 0, the optimum /(oj) tends to a constant. 
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VIII. THE ROLE OF NONEXCESS BANDWIDTH SYSTEMS 

In the previous sections we have determined the optimum trans- 
mitter under the hypothesis that the system is nonexcess bandwidth. 
Here we point out that this hypothesis is not very restrictive. 

In systems in which the transmitter is required to be passband, it 
follows, under very mild assumptions on the channel characteristics, 
that the optimum transmitter (subject to an output power constraint) 
is a nonexcess bandwidth system. The mathematical proof of the 
optimality of the nonexcess bandwidth system is considered in detail 
in Appendix A. For an example, if for each u(\a\ < w/T) 

|Ci(») + #.(•) | > ki(«+?^) + iC 2 (a> + ^)| ( fc ^°)- 

then the optimal transmitter has no energy outside 

| CO — 2tt/o| ^ jj 

For systems allowing any matrix transmitter, the question arises 
whether or not the optimal transmitter is passband. If the answer to 
the question is negative, the next question is whether or not the opti- 
mal transmitter is nonexcess bandwidth. The answers to these ques- 
tions depend on the system parameters, and there are channels for 
which the answers to both questions are negative. It is beyond the 
scope of this paper to give a detailed mathematical discussion of these 
more complex systems. Such systems are still under investigation, and 
so we shall limit ourselves to mentioning without proof some important 
facts concerning the analysis of such systems. 

The analysis begins by returning to Section IV fixing w and posing 
the extremal problem of 

max det (£ GtCtC k G k + NJ), 

subject to tr £ G\G k = /• If for each u> it is optimal to expend all of 
/ on one of the Gk's, then we are in the line pursued in the previous sec- 
tions. However, to achieve optimality one may need to use more than 
one G. Indeed, H. Witsenhausen has solved this determinant extremal 
problem showing that at most two G k 's are required to achieve opti- 
mality, and there are instances where two GVs are necessary. Even 
when two Gk's are needed, the w matrix remains a scalar matrix and 
once again the trace and the determinant optimization are equivalent. 
The fact that two Gk's are required means the transmitter is excess 
bandwidth. 
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Witsenhausen has shown that when two GVs are needed, one can 
be taken to be a multiple of f . J J and the other a multiple of 

( _i)- A ^hough both G k 's cannot have the passband form, the 

( • _ i ) ma trix corresponds to a very simple structural variation of 

a passband filter. 

We mention in closing that systems whose optimization takes us 
outside the realm of passband structures can be analyzed via equivalent 
canonical nonexcess bandwidth passband systems. The equivalence is 
in the sense that MMSE versus P curves for the two systems are 
identical, and optimum design can be carried out in the canonical 
system and then transformed to the more complicated system. 
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APPENDIX A 

Optimality of a Nonexcess Bandwidth System 

Fix P > and q(u>) a positive continuous real function on 
(— °°, + °°). In the text we are confronted with the optimization 

where the sup is over all nonnegative Lebesque integrable r(») for which 

•+oo 



i 



r(a)du ^ P > 0. 



We show here that, under weak conditions on <?(u>), the optimization 
problem can be replaced by an equivalent "nonexcess bandwidth 
problem," namely, find 

r+rlT 
sup / log [r(e))q(u) + IJdta, 

J-r/T 

where g(o>) is a given continuous function and f (ta) is any nonnegative 
integrable function satisfying 



/ 



t/t 

f(u)du = P > 0. 

IT 
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Define g(«) on [— v/T, ir/T] to be the envelope sup q(u + 2-irk/T). 
To avoid annoying pathologies, assume g(o>) is such that for each cu 



k\ q{<* + -jrj - £(«) 



is not empty. Moreover, assume that (— ir/T, tt/jT) can be expressed 
as a disjoint union of subsets {F m }f of total measure 27I-/7 7 such that 
on each V m there exists a fc m so 

/ 27rA; m \ , N 

q V ~~ IF/ = 5(c ^ 

holds uniformly in a> on V m . So 5(a)) is continuous on ( — t/T, t/T). 
Define 

Given any r(w) ^ satisfying ||r||i = P, define p on (- co, 00) by 
/ ( 2irfr w \_ [fc r ("+ffi for^GF m w-1,2, - - , Af 



So 



f prfw = f) / p f to + -^r J dw 

where the second equality results from the definition of p and the third 
equality is from the Lebesque Dominated Convergence Theorem. Now 
for |w| < t/T 

, . _ / , 2irk \ ~ ( , 2irk\ ( , 2irfc \ 

where the very last equality follows from the fact that p vanishes off 
V. Since in L[— ir/T, t/T~\p(u) always fares at least as well as r(o>), 
we have the fact that the supremum can be taken over the class of 
nonnegative functions vanishing off V. 

In the applications it often occurs that V C (-t/T, t/T), in which 
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case q(o)) = g(w) on V and the optimization problem becomes 

•t/t 



sup / 

\\r\\=P J- 



log [r(w)g(oj) + l]dco. 

-T/T 



Even when 7 CjZ ( — ir/T, tt/T), we need only solve 

r/T 
llr||=P •'-'Z 2 ' 



r*IT 

sup / log [r(a>)g(w) 4- l]d« 

llfll = p J -tIT 



with the optimand "rearranged" to produce the desired r{w). The re- 
arrangement procedure is simply that, for each a> G ( — x/7 7 , 7r/!T), we 
define r(a> + 2wk m /T) = f(a»). Elsewhere r(w) is defined to be zero. 
In dealing with even q(o>), if V <£ (-ir/T, tt/T), the rearrangement 
produces an uneven r(co). When this occurs, [>(&>) + r( — o>)]/2 pro- 
vides an even optimand. 

APPENDIX B 

Maximization of the Exponent Functional 

Let q(w) be a continuous positive function on an interval [a, 6], 
Fixing a real number P > 0, let T be the convex set of nonnegative 
continuous functions with integral less than or equal to P. We seek y 
to maximize the nonlinear functional 



I(y) = flog (I + 79 ). 

J a 



This same problem occurs in classical information theory where, for 
reasons we shall see, it is dubbed "the water-pouring problem." 
Although the solution is correctly described in the literature, the sup- 
porting arguments are formal (for example, see Ref. 16 or 17). We 
give a rigorous proof here, although our argument is not construc- 
tive in that the extremal function is "pulled out of the air." To moti- 
vate the extremal function, the reader can turn to the references or 
supply for himself a variational derivation. 

Now 7(7) is concave on r, as we see by employing the Liebnitz rule 
to confirm the strict negativity of I"[Xyi + (1 — X)7 2 ] on ^ X ^ 1 
with 71 and 72 in r (differentiation is with respect to X). It is clear that 
if the extremal function exists it has integral equal to P and so we can 
redefine r to require equality of the integrals. 

For each constant e, the function (e — q~ 1 ) + denotes the function 
equal to e — q' 1 when e - q~ l > and equal to zero otherwise. Now 
f (C — g _1 ) is a continuous strictly increasing function of Q with range 
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\ ' ' ' 

\ 



\ 



\ 




Fig. 7 — Optimal power spectral density. 



[0, * ]. Fix 6 so f{Q — q~ l )+ = P and call the resulting function 7. 
To show 7(f) is the global maximum of I (7) over T, let 71 denote 
any other function in T and let us investigate the segment 
{Xf + (1 — X)7i, ^ X ^ 1}. Now /[X7 + (1 — X)7i] is concave in 
X and straightforwardly 



Z'[>7 + (l - x) 7 i]|x=i = e- 1 (p - f Qqyi - f 



7i 



which is nonnegative as Qq ^ 1. By definition, for a concave function 
the graph lies above any chord joining two points on the graph. So 
X = 1 must be a point of global maxima of the segment. 

Also, 7 is the unique point of maxima since, if there were another 
point of maxima 7, we would have I(y) constant on the line segment 
joining 7 and 7 contradicting the strict negativity of /". 

To understand the water-pouring terminology, look at Fig. 7 where 
we consider the graph of q~ l with vertical walls based at [a, <? -1 ( a )!l and 
[b, g -1 (&)3 to be a vessel into which water of amount (area) P is poured. 
Relocate the co axis to the water level line. Then reflecting the water 
accumulation about the level line gives the shape of 7. 

We mention in closing that 7 is optimal in a larger set than T ob- 
tained by requiring integrability rather than continuity in the defini- 
tion of the constraint set. The optimality over the larger set follows 
from a function space continuity argument. 



REFERENCES 



1. MacColl, L. A., "Signaling Method and Apparatus," U. S. Patent No. 2,056,284, 
October 6, 1936. 



DFE IN QAM SYSTEMS 1849 

2. Austin, M. E., "Decision — Feedback Equalization for Digital Communication 

over Dispersive Channels," Technical Report 461, Research Laboratory of 
Electronics, M.I.T., August 11, 1967. 

3. Keeler, R. J., "Construction and Evaluation of a Decision Feedback Equalizer," 

Record of IEEE Int. Conf. on Comm. 1971, Montreal, Canada, June 1971, 
pp. 21-8 to 21-18. 

4. George, D. A., Bowen, R. R., and Storey, J. R., "An Adaptive Decision Feedback 

Equalizer," IEEE Trans. Comm. Tech., COM-19, June 1971, pp. 281-293. 

5. Monsen, P., "Feedback Equalization for Fading Dispersive Channels," IEEE 

Trans, on Info. Theory, IT-17, January 1971, pp. 56-64. 

6. Price, R., "Nonlinearly Feedback-Equalized PAM vs. Capacity for Noisy Filter 

Channels," Record of IEEE Int. Conf. on Comm. 1972, Philadelphia, Pa., 
June 1972, pp. 22-12 to 22-17. 

7. Salz, J., "Optimum Mean-Square Decision Feedback Equalization," B.S.T.J., 

52, No. 8 (October 1973), pp. 1341-1373. 

8. Lucky, R. W., Salz, J., and Weldon, E. S., Jr., Principles of Data Communication, 

New York : McGraw-Hill, 1968. 

9. Kobayashi, H., "Simultaneous Adaptive Estimation and Decision Algorithm for 

Carrier- Modultated Data Transmission Systems," IEEE Trans. Comm. 
Tech., COM-19, October 1971, pp. 835-848. 

10. Dugundji, J., "Envelopes and Pre-Envelopes of Real Waveforms," IRE Trans. 

Info. Theory, IT-/,, March 1958, pp. 53-57. 

11. Wozencraft, J. M., and Jacobs, I. M., Principles of Communication Engineering, 

New York: John Wiley and Sons, 1965, pp. 492-504. 

12. Vainberg, M. M., Variational Methods for the Study of Nonlinear Operators, 

San Francisco : Holden-Day, Chapter III. 

13. Dunford, N., and Schwartz, J., Linear Operators, Part III, New York: Wiley- 

Interscience, 1971, Chapter XV. 

14. Wiener, N., and Akutowicz, E. J., "A Factorization of Positive Hermitian 

Matrices," J. Math, and Mech., 8, 1959, pp. 111-120. 

15. Doob, J. L., Stochastic Processes, New York: John Wiley and Sons, 1967, pp. 

160-164. 

16. Fano, R. M., Trans?nission of Information, New York: John Wiley and Sons, 

1961, pp. 168-178. 

17. Walvick, E. A., "On the Capacity of an Ensemble of Channels with Differing 

Parameters," B.S.T.J., 49, No. 3 (March 1970), pp. 415^29. 



